Sancov guard draft #7

ramosian-glider · 2025-02-17T15:27:49Z

No description provided.

kcov used to obey clang-format style, but somehow diverted over time. No functional change. Signed-off-by: Alexander Potapenko <[email protected]>

Group several kcov-related fields (area, size, mode, sequence) that are stored in various structures, into `struct kcov_state`, so that these fields can be easily passed around and manipulated. This prepares us for the upcoming change that will introduce more kcov state. Signed-off-by: Alexander Potapenko <[email protected]>

kernel/kcov.c

jiangenj · 2025-02-20T07:27:12Z

Do you have the userspace prog to test now?
Also I doubt how sancov_guards for modules will be loaded.

ramosian-glider · 2025-02-21T16:51:59Z

Here you go:
kcov.txt

(rename from kcov.txt to kcov.c)

As for modules, I haven't looked closely, but I don't expect any issues with those.

include/asm-generic/vmlinux.lds.h

kernel/kcov.c

The new config switches coverage instrumentation to using __sanitizer_cov_trace_pc_guard(u32 *guard) instead of __sanitizer_cov_trace_pc(void) Each callback receives a unique 32-bit guard variable residing in the __sancov_guards section. Those guards can be used by kcov to deduplicate the coverage on the fly. As a first step, we make the new instrumentation mode 1:1 compatible with the old one. Signed-off-by: Alexander Potapenko <[email protected]>

Keep kcov_state.area as the pointer to the memory buffer used by kcov and shared with the userspace. Store the pointer to the trace (part of the buffer holding sequential events) separately, as we will be splitting that buffer in multiple parts. No functional change so far. Signed-off-by: Alexander Potapenko <[email protected]>

ramosian-glider · 2025-03-04T09:54:04Z

To simplify the API and do not require the user to unconditionally carve out the bitmap from the mmapped region, we decided to remove the buffer partitioning API and introduce ioctl(KCOV_UNIQUE_ENABLE).
Now it's possible to use the old code, but do one-off runs with deduplicated coverage if needed.

kernel/kcov.c

jiangenj · 2025-03-04T12:43:26Z

kernel/kcov.c

+		this_cpu_write(saved_index, index);
+		return old_guard;
+	}
+	return index;


in previous version, userspace gets pos number as 177 and actual printed cover (not 0) is 22.
Curious to know in this version, how many actual covered pc / pos?
Please share the updated userspace program too.

I am not sure I understand the question, can you please elaborate?

Please share the updated userspace program too.

Please check the updated documentation, you basically need to replace KCOV_ENABLE with KCOV_UNIQUE_ENABLE and memset the bitmap with zeroes every time you want to reset the coverage.

kcov-new.txt

Here is the updated userspace program.

For example,

./kcov-guard cover[0]: cea cover[1]: ffffffff81c6472a cover[2]: ffffffff81c647de cover[3]: ffffffff81c6481e cover[4]: ffffffff81c649a7 cover[5]: ffffffff81b54e8b cover[6]: ffffffff81b4ee3b cover[7]: ffffffff81b4eeb9 cover[8]: ffffffff81b4ef32 cover[9]: ffffffff81b4ef67 cover[10]: ffffffff81b54edf cover[11]: ffffffff81b54f13 cover[12]: ffffffff81b54fcb cover[13]: ffffffff823c3a04 cover[14]: ffffffff823c37aa cover[15]: ffffffff82461107 cover[16]: ffffffff824612a0 cover[17]: ffffffff823c3869 cover[18]: ffffffff81c64cd9 cover[19]: ffffffff81c64d39 cover[20]: ffffffff81c63d8d cover[21]: ffffffff81c63dea cover[22]: ffffffff81883de0 cover[23]: ffffffff81883e8f cover[24]: ffffffff81c6472a cover[25]: ffffffff81c647de cover[26]: ffffffff81c6481e cover[27]: ffffffff81c649a7 cover[28]: ffffffff81b54e8b cover[29]: ffffffff81b4ee3b cover[30]: ffffffff81b4eeb9 cover[31]: ffffffff81b4ef32 cover[32]: ffffffff81b4ef67 cover[33]: ffffffff81b54edf cover[34]: ffffffff81b54f13 cover[35]: ffffffff81b54fcb cover[36]: ffffffff823c3a04 cover[37]: ffffffff823c37aa cover[38]: ffffffff82461107 cover[39]: ffffffff824612a0 cover[40]: ffffffff823c3869 cover[41]: ffffffff81c64cd9 cover[42]: ffffffff81c64d39 cover[43]: ffffffff81c63d8d cover[44]: ffffffff81c63dea cover[45]: ffffffff81883de0 cover[46]: ffffffff81883e8f

cover[0] is the pos number which is 0xcea, while actually covered number is 46, so actual used index is only 46/0xcea =1.4%.

The same for uniq:

./kcov-guard uniq is 23/176=13.1%. cover[0]: b0 cover[1]: ffffffff81c6472a cover[2]: ffffffff81c647de cover[3]: ffffffff81c6481e cover[4]: ffffffff81c649a7 cover[5]: ffffffff81b54e8b cover[6]: ffffffff81b4ee3b cover[7]: ffffffff81b4eeb9 cover[8]: ffffffff81b4ef32 cover[9]: ffffffff81b4ef67 cover[10]: ffffffff81b54edf cover[11]: ffffffff81b54f13 cover[12]: ffffffff81b54fcb cover[13]: ffffffff823c3a04 cover[14]: ffffffff823c37aa cover[15]: ffffffff82461107 cover[16]: ffffffff824612a0 cover[17]: ffffffff823c3869 cover[18]: ffffffff81c64cd9 cover[19]: ffffffff81c64d39 cover[20]: ffffffff81c63d8d cover[21]: ffffffff81c63dea cover[22]: ffffffff81883de0 cover[23]: ffffffff81883e8f

In sparse mode, it didn't print cover[0], cover[1];

./kcov-guard sparse cover[2]: ffffffff81c64cd9 ... cover[176]: ffffffff82426910 cover[177]: ffffffff824242e6

After add printf for scan_size, it reports

scan_size=65536

So, it means overflow here. For 177 covered pc and kcov overflow happened.

ioctl(KCOV_UNIQUE_ENABLE) enables collection of deduplicated coverage in the presence of CONFIG_KCOV_ENABLE_GUARDS. The buffer shared with the userspace is divided in two parts, one holding a bitmap, and the other one being the trace. The single parameter of ioctl(KCOV_UNIQUE_ENABLE) determines the number of words used for the bitmap. Each __sanitizer_cov_trace_pc_guard() instrumentation hook receives a pointer to a unique guard variable. Upon the first call of each hook, the guard variable is initialized with a unique integer, which is used to map those hooks to bits in the bitmap. In the new coverage collection mode, the kernel first checks whether the bit corresponding to a particular hook is set, and then, if it is not, the PC is written into the trace buffer, and the bit is set. Also update the documentation. Signed-off-by: Alexander Potapenko <[email protected]>

jiangenj · 2025-03-10T04:26:18Z

Here you go: kcov.txt

(rename from kcov.txt to kcov.c)

As for modules, I haven't looked closely, but I don't expect any issues with those.

For example, If i load the module when trace_pc_guard enabled

 insmod ./usbmon.ko
[ 3686.608008][ T9399] module: usbmon: Unknown rela relocation: 42
insmod: ERROR: could not insert module ./usbmon.ko: Invalid module format

The readelf shows it has many __sancov_guards sections

...
  [294] __sancov_guards   PROGBITS         0000000000000000  00011ab8
  [295] __sancov_guards   PROGBITS         0000000000000000  00011ac8
  [296] __sancov_guards   PROGBITS         0000000000000000  00011acc

If I made the similar change to vmlinux.lds.S to module.lds.S. Then I can see

readelf -S /local/mnt/workspace/jiangenj/syzkaller/x86_64/linux/drivers/usb/mon/usbmon.ko|grep sancov
  [ 7] __sancov_guards   NOBITS           0000000000000000  00001460
  [52] .text.sancov[...] PROGBITS         0000000000000000  00003490

However, load the module, I see see

# insmod ./usbmon.ko
[  115.752127][ T9282] module: usbmon: Unknown rela relocation: 42
insmod: ERROR: could not insert module ./usbmon.ko: Invalid module format

jiangenj · 2025-03-10T07:28:54Z

Here you go: kcov.txt
(rename from kcov.txt to kcov.c)
As for modules, I haven't looked closely, but I don't expect any issues with those.

For example, If i load the module when trace_pc_guard enabled
 insmod ./usbmon.ko
[ 3686.608008][ T9399] module: usbmon: Unknown rela relocation: 42
insmod: ERROR: could not insert module ./usbmon.ko: Invalid module format
The readelf shows it has many __sancov_guards sections
...
  [294] __sancov_guards   PROGBITS         0000000000000000  00011ab8
  [295] __sancov_guards   PROGBITS         0000000000000000  00011ac8
  [296] __sancov_guards   PROGBITS         0000000000000000  00011acc
If I made the similar change to vmlinux.lds.S to module.lds.S. Then I can see
readelf -S /local/mnt/workspace/jiangenj/syzkaller/x86_64/linux/drivers/usb/mon/usbmon.ko|grep sancov
  [ 7] __sancov_guards   NOBITS           0000000000000000  00001460
  [52] .text.sancov[...] PROGBITS         0000000000000000  00003490
However, load the module, I see see
# insmod ./usbmon.ko
[  115.752127][ T9282] module: usbmon: Unknown rela relocation: 42
insmod: ERROR: could not insert module ./usbmon.ko: Invalid module format

Add some search results:
In llvm 42 is https://github.com/microsoft/llvm/blob/master/include/llvm/BinaryFormat/ELFRelocs/x86_64.def#L45C11-L45C33 R_X86_64_REX_GOTPCRELX
In objdump -r shows

RELOCATION RECORDS FOR [.text.sancov.module_ctor_trace_pc_guard]:
OFFSET           TYPE                     VALUE
0000000000000007 R_X86_64_REX_GOTPCRELX   __start___sancov_guards-0x4
000000000000000e R_X86_64_REX_GOTPCRELX   __stop___sancov_guards-0x4

ramosian-glider added 2 commits February 7, 2025 14:13

kcov: apply clang-format to kernel/kcov.c and include/linux/kcov.h

e125a66

kcov used to obey clang-format style, but somehow diverted over time. No functional change. Signed-off-by: Alexander Potapenko <[email protected]>

jiangenj reviewed Feb 18, 2025

View reviewed changes

kernel/kcov.c Show resolved Hide resolved

jiangenj reviewed Feb 25, 2025

View reviewed changes

include/asm-generic/vmlinux.lds.h Show resolved Hide resolved

quic-likaid reviewed Feb 25, 2025

View reviewed changes

kernel/kcov.c Show resolved Hide resolved

ramosian-glider force-pushed the sancov-guard-take2 branch from 43b9bbb to dc780c7 Compare March 4, 2025 08:33

ramosian-glider added 2 commits March 4, 2025 10:14

ramosian-glider force-pushed the sancov-guard-take2 branch 2 times, most recently from 127e5b3 to 5cba390 Compare March 4, 2025 09:47

ramosian-glider force-pushed the sancov-guard-take2 branch from 5cba390 to 8543e55 Compare March 4, 2025 12:01

jiangenj reviewed Mar 4, 2025

View reviewed changes

kernel/kcov.c Outdated Show resolved Hide resolved

jiangenj reviewed Mar 4, 2025

View reviewed changes

ramosian-glider force-pushed the sancov-guard-take2 branch from 8543e55 to 91a94ce Compare March 6, 2025 13:27

ramosian-glider force-pushed the sancov-guard-take2 branch from 91a94ce to ab8adf7 Compare March 6, 2025 13:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sancov guard draft #7

Sancov guard draft #7

ramosian-glider commented Feb 17, 2025

jiangenj commented Feb 20, 2025

ramosian-glider commented Feb 21, 2025

ramosian-glider commented Mar 4, 2025

jiangenj Mar 4, 2025

ramosian-glider Mar 6, 2025

ramosian-glider Mar 6, 2025

jiangenj Mar 10, 2025

jiangenj commented Mar 10, 2025

jiangenj commented Mar 10, 2025

Sancov guard draft #7

Are you sure you want to change the base?

Sancov guard draft #7

Conversation

ramosian-glider commented Feb 17, 2025

jiangenj commented Feb 20, 2025

ramosian-glider commented Feb 21, 2025

ramosian-glider commented Mar 4, 2025

jiangenj Mar 4, 2025

Choose a reason for hiding this comment

ramosian-glider Mar 6, 2025

Choose a reason for hiding this comment

ramosian-glider Mar 6, 2025

Choose a reason for hiding this comment

jiangenj Mar 10, 2025

Choose a reason for hiding this comment

jiangenj commented Mar 10, 2025

jiangenj commented Mar 10, 2025